Bayesian Additive Regression Trees Research Articles

Variable Selection Using Bayesian Additive Regression Trees

Variable Selection Using Bayesian Additive Regression Trees

Assessing the Causal Effect of Special Education Services on Math Achievement: A Causal Inference and Machine Learning Study

This study aims to assess the Average Treatment Effect (ATE) of receiving special education services on revised Item Response Theory (IRT) scaled math achievement test scores. By employing a methodological repertoire comprising linear regression with ordinary least squares (OLS), propensity score matching (PSM), Bayesian Additive Regression Trees (BART), and Multilayer Perceptron (MLP), we examine the impact of these interventions. Leveraging data from the Early Childhood Longitudinal Study Kindergarten 2010-11 cohort (ECLS-K:2011), we systematically analyze the ATE of special education services on students' math achievement. The results show that all models yield negative ATE results, suggesting a deleterious effect of special education services on fifth-grade math scores. Furthermore, we employ Principal Component Analysis (PCA) to corroborate these findings, aligning with outcomes obtained from causal inference and Machine Learning (ML) based methods. This research emphasizes the importance of method diversity in educational research and highlights the need for assessments of intervention effectiveness to help educational practices and policies.

Modelling the potential range of Agrilus planipennis in Europe according to current and future climate conditions

Agrilus planipennis, the emerald ash borer, is a species native to East Asia that was accidentally introduced to North America and Eastern Europe. In North America, it is responsible for tremendous damage. In Europe, its range has quickly expanded from the east where it was introduced in 2003, and it threatens the species of the genus Fraxinus. We developed an ensemble modelling approach to model the potential range of A. planipennis according to current climate conditions and four scenarios of climate change: SSP1–2.6, SSP2–4.5, SSP3–7.0 SSP5–8.5 in the period 2041–2060. We used three algorithms; random forest, boosted regression trees and Bayesian additive regression trees with occurrence data from both native and invaded ranges. The results indicate that most of the European continent is climatically suitable for A. planipennis. In Western Europe, the northern limit of the range is located in the British Isles and southern Scandinavia. The projection of the models according to estimates of future climate conditions shows that climate suitability would mostly remain unchanged in 2041–2060. During that period, the potential range is expected to slightly shrink in the south, around the Mediterranean Basin, and expand at its northern limit. Our results confirm that A. planipennis is, and will remain, a major threat to forest and ornamental ash tree health across Europe.

Partially fixed bayesian additive regression trees

Bayesian Additive Regression Trees (BART) is a widely popular nonparametric regression model known for its accurate prediction capabilities. In certain situations, there is knowledge suggesting the existence of certain dominant variables. However, the BART model fails to fully utilize the knowledge. To tackle this problem, the paper introduces a modification to BART known as the Partially Fixed BART model. By fixing a portion of the trees' structure, this model enables more efficient utilization of prior knowledge, resulting in enhanced estimation accuracy. Moreover, the Partially Fixed BART model can offer more precise estimates and valuable insights for future analysis even when such prior knowledge is absent. Empirical results substantiate the enhancement of the proposed model in comparison to the original BART.

Demonstrating the utility of Instrumented Gait Analysis in the treatment of children with cerebral palsy.

Instrumented gait analysis (IGA) has been around for a long time but has never been shown to be useful for improving patient outcomes. In this study we demonstrate the potential utility of IGA by showing that machine learning models are better able to estimate treatment outcomes when they include both IGA and clinical (CLI) features compared to when they include CLI features alone. We carried out a retrospective analysis of data from ambulatory children diagnosed with cerebral palsy who were seen at least twice at our gait analysis center. Individuals underwent a variety of treatments (including no treatment) between sequential gait analyses. We fit Bayesian Additive Regression Tree (BART) models that estimated outcomes for mean stance foot progression to demonstrate the approach. We built two models: one using CLI features only, and one using CLI and IGA features. We then compared the models' performance in detail. We performed similar, but less detailed, analyses for a number of other outcomes. All results were based on independent test data from a 70%/30% training/testing split. The IGA model was more accurate than the CLI model for mean stance-phase foot progression outcomes (RMSEIGA = 11∘, RMSECLI = 13∘) and explained more than 1.5 × as much of the variance (R2IGA = .45, R2CLI = .28). The IGA model outperformed the CLI model for every level of treatment complexity, as measured by number of simultaneous surgeries. The IGA model also exhibited superior performance for estimating outcomes of mean stance-phase knee flexion, mean stance-phase ankle dorsiflexion, maximum swing-phase knee flexion, gait deviation index (GDI), and dimensionless speed. The results show that IGA has the potential to be useful in the treatment planning process for ambulatory children diagnosed with cerebral palsy. We propose that the results of machine learning outcome estimators-including estimates of uncertainty-become the primary IGA tool utilized in the clinical process, complementing the standard medical practice of conducting a through patient history and physical exam, eliciting patient goals, reviewing relevant imaging data, and so on.

Predicting post-treatment symptom severity for adults receiving psychological therapy in routine care for generalised anxiety disorder: a machine learning approach

Approximately half of generalised anxiety disorder (GAD) patients do not recover from first-line treatments, and no validated prediction models exist to inform individuals or clinicians of potential treatment benefits. This study aimed to develop and validate an accurate and explainable prediction model of post-treatment GAD symptom severity. Data from adults receiving treatment for GAD in eight Improving Access to Psychological Therapies (IAPT) services (n=15,859) were separated into training, validation and holdout datasets. Thirteen machine learning algorithms were compared using 10-fold cross-validation, against two simple clinically relevant comparison models. The best-performing model was tested on the holdout dataset and model-specific explainability measures identified the most important predictors. A Bayesian Additive Regression Trees model out-performed all comparison models (MSE=16.54[95%CI=15.58;17.51]; MAE=3.19; R²=0.33, including single predictor linear regression: MSE=20.70[95%CI=19.58;21.82]; MAE=3.94; R²=0.14). The five most important predictors were: PHQ-9 anhedonia, GAD-7 annoyance/irritability, restlessness and fear items, then the referral-assessment waiting time. The best-performing model accurately predicted post-treatment GAD severity using only pre-treatment data, outperforming comparison models that approximated clinical judgement and remaining within the GAD-7 error of measurement and minimal clinically important differences. This model could inform treatment decision-making and provide desired information to clinicians and patients receiving treatment for GAD.

Generalized Bayesian Additive Regression Trees Models: Beyond Conditional Conjugacy

Bayesian additive regression trees have seen increased interest in recent years due to their ability to combine machine learning techniques with principled uncertainty quantification. The Bayesian backfitting algorithm used to fit BART models, however, limits their application to a small class of models for which conditional conjugacy exists. In this article, we greatly expand the domain of applicability of BART to arbitrary generalized BART models by introducing a very simple, tuning-parameter-free, reversible jump Markov chain Monte Carlo algorithm. Our algorithm requires only that the user be able to compute the likelihood and (optionally) its gradient and Fisher information. The potential applications are very broad; we consider examples in survival analysis, structured heteroskedastic regression, and gamma shape regression.

Implementation of Proximal and Remote Soil Sensing, Data Fusion and Machine Learning to Improve Phosphorus Spatial Prediction for Farms in Ontario, Canada

One of the challenges in site-specific phosphorus (P) management is the substantial spatial variability in plant available P across fields. To overcome this barrier, emerging sensing, data fusion, and spatial predictive modeling approaches are needed to accurately reveal the spatial heterogeneity of P. Seven spatially variable fields located in Ontario, Canada are clustered into two zones; four fields are located in eastern Ontario and three others are located in western Ontario. This study compares Bayesian Additive Regression Trees (BART), Support Vector Machine regressor (SVM), and Ordinary Kriging (OK), along with novel data fusion concepts, to analyze integrated high-density spatial data layers related to spatial variability in soil available P. Feature selection and interaction detection using BART variable selection and Recursive Feature Elimination (RFE) for SVM were applied to 42 predictors, including soil-vegetation indices derived from PlanetScope multispectral imagery, high-density apparent soil electrical conductivity (ECa), and high-resolution topographic attributes derived from DUALEM-21S and a Real-Time Kinematic (RTK) global navigation satellite systems (GNSS) receiver, respectively. Modeling spatial heterogeneity of soil available P with BART showed higher accuracy than SVM and OK in both zones of this study when trained and tested on ground truth data from clusters of farms. A BART variable selection approach resulted in six auxiliary predictors of soil available P in the eastern zone, while only four predictors were selected to predict P in the western zone. RFE for SVM resulted in models with 15 and 12 auxiliary predictors in the eastern and western Ontario zones. Topographic elevation was the most influential predictor of soil available P in both zones. Compared with the SVM and OK methods, BART exhibited lower average RMSE values for individual fields of 1.86 ppm and 3.58 ppm across the eastern and western Ontario zones, respectively, along with higher R2 values of 0.85 and 0.83, respectively. In contrast, SVM had RMSE values for individual fields in the eastern and western Ontario zones, respectively, averaging 5.04 ppm and 7.51 ppm and R2 values of 0.27 and 0.43. RMSE values for soil available P in individual fields across the eastern and western Ontario zones averaged 4.77 ppm and 7.81 ppm, respectively, with the OK method, while R2 values averaged 0.19 and 0.44. The selection of suitable auxiliary predictors and data fusion, combined with BART spatial machine learning algorithms, have potential to be a useful tool to accurately estimate spatial patterns in soil available P for agricultural fields in Ontario, Canada.

A comparison of Bayesian and frequentist approaches to incorporating clinical and biological information for the prediction of response to standardized pediatric colitis therapy.

The prospective cohort study PROTECT is the largest study in pediatric ulcerative colitis (UC) with standardized treatments, providing valuable data for predicting clinical outcomes. PROTECT and previous studies have identified characteristics associated with clinical outcomes. In this study, we aimed to compare predictive modeling between Bayesian analysis including machine learning and frequentist analysis. The key outcomes for this analysis were week 4, 12 and 52 corticosteroid (CS)-free remission following standardized treatment from diagnosis. We developed predictive modeling with multivariable Bayesian logistic regression (BLR), Bayesian additive regression trees (BART) and frequentist logistic regression (FLR). The effect estimate of each risk factor was estimated and compared between the BLR and FLR models. The predictive performance of the models was assessed including area under curve (AUC) of the receiver operating characteristic (ROC) curve. Ten-fold cross-validation was performed for internal validation of the models. The estimation contained 95% credible (or confidence) interval (CI). The statistically significant associations between the risk factors and early or late outcomes were consistent between all BLR and FLR models. The model performance was similar while BLR and BART models had narrower credible intervals of AUCs. To predict week 4 CS-free remission, the BLR model had AUC of 0.69 (95% CI 0.67-0.70), the BART model had AUC of 0.70 (0.67-0.72), and the FLR had AUC of 0.70 (0.65-0.76). To predict week 12 CS-free remission, the BLR model had AUC of 0.78 (0.77-0.79), the BART model had AUC of 0.78 (0.77-0.79), and the FLR model had AUC of 0.79 (0.74-0.83). To predict week 52 CS-free remission, the BLR model had AUC of 0.69 (0.68-0.70), the BART model had AUC of 0.69 (0.67-0.70), and the FLR model had AUC of 0.69 (0.64-0.74). The BART model identified nonlinear associations. BLR and BART models had intuitive interpretation on interval estimation, better precision in estimating the AUC and can be alternatives for predicting clinical outcomes in pediatric patients with UC. BART model can estimate nonlinear nonparametric association.

Investigating Growth-at-Risk Using a Multicountry Nonparametric Quantile Factor Model

We develop a nonparametric quantile panel regression model. Within each quantile, the quantile function is a combination of linear and nonlinear parts, which we approximate using Bayesian Additive Regression Trees (BART). Cross-sectional information is captured through a conditionally heteroscedastic latent factor. The nonparametric feature enhances flexibility, while the panel feature increases the number of observations in the tails. We develop Bayesian methods for inference and apply several versions of the model to study growth-at-risk dynamics in a panel of 11 advanced economies. Our framework usually improves upon single-country quantile models in recursive growth forecast comparisons.

Extinction risk predictions for the world's flowering plants to support their conservation.

More than 70% of all vascular plants lack conservation status assessments. We aimed to address this shortfall in knowledge of species extinction risk by using the World Checklist of Vascular Plants to generate the first comprehensive set of predictions for a large clade: angiosperms (flowering plants, c. 330 000 species). We used Bayesian Additive Regression Trees (BART) to predict the extinction risk of all angiosperms using predictors relating to range size, human footprint, climate, and evolutionary history and applied a novel approach to estimate uncertainty of individual species-level predictions. From our model predictions, we estimate 45.1% of angiosperm species are potentially threatened with a lower bound of 44.5% and upper bound of 45.7%. Our species-level predictions, with associated uncertainty estimates, do not replace full global, or regional Red List assessments, but can be used to prioritise predicted threatened species for full Red List assessment and fast-track predicted non-threatened species for Least Concern assessments. Our predictions and uncertainty estimates can also guide fieldwork, inform systematic conservation planning and support global plant conservation efforts and targets.

A BAYESIAN MACHINE LEARNING APPROACH FOR ESTIMATING HETEROGENEOUS SURVIVOR CAUSAL EFFECTS: APPLICATIONS TO A CRITICAL CARE TRIAL.

Assessing heterogeneity in the effects of treatments has become increasingly popular in the field of causal inference and carries important implications for clinical decision-making. While extensive literature exists for studying treatment effect heterogeneity when outcomes are fully observed, there has been limited development in tools for estimating heterogeneous causal effects when patient-centered outcomes are truncated by a terminal event, such as death. Due to mortality occurring during study follow-up, the outcomes of interest are unobservable, undefined, or not fully observed for many participants in which case principal stratification is an appealing framework to draw valid causal conclusions. Motivated by the Acute Respiratory Distress Syndrome Network (ARDSNetwork) ARDS respiratory management (ARMA) trial, we developed a flexible Bayesian machine learning approach to estimate the average causal effect and heterogeneous causal effects among the always-survivors stratum when clinical outcomes are subject to truncation. We adopted Bayesian additive regression trees (BART) to flexibly specify separate mean models for the potential outcomes and latent stratum membership. In the analysis of the ARMA trial, we found that the low tidal volume treatment had an overall benefit for participants sustaining acute lung injuries on the outcome of time to returning home but substantial heterogeneity in treatment effects among the always-survivors, driven most strongly by biologic sex and the alveolar-arterial oxygen gradient at baseline (a physiologic measure of lung function and degree of hypoxemia). These findings illustrate how the proposed methodology could guide the prognostic enrichment of future trials in the field.

Does crouch alter the effects of neuromuscular impairments on gait? A simulation study

Cerebral palsy (CP) is a neurologic injury that impacts control of movement. Individuals with CP also often develop secondary impairments like weakness and contracture. Both altered motor control and secondary impairments influence how an individual walks after neurologic injury. However, understanding the complex interactions between and relative effects of these impairments makes analyzing and improving walking capacity in CP challenging. We used a sagittal-plane musculoskeletal model and neuromuscular control framework to simulate crouch and nondisabled gait. We perturbed each simulation by varying the number of synergies controlling each leg (altered control), and imposed weakness and contracture. A Bayesian Additive Regression Trees (BART) model was also used to parse the relative effects of each impairment on the muscle activations required for each gait pattern. By using these simulations to evaluate gait-pattern specific effects of neuromuscular impairments, we identified some advantages of crouch gait. For example, crouch tolerated 13 % and 22 % more plantarflexor weakness than nondisabled gait without and with altered control, respectively. Furthermore, BART demonstrated that plantarflexor weakness had twice the effect on total muscle activity required during nondisabled gait than crouch gait. However, crouch gait was also disadvantageous in the presence of vasti weakness: crouch gait increased the effects of vasti weakness on gait without and with altered control. These simulations highlight gait-pattern specific effects and interactions between neuromuscular impairments. Utilizing computational techniques to understand these effects can elicit advantages of gait deviations, providing insight into why individuals may select their gait pattern and possible interventions to improve energetics.

The impact of early childhood education and medical care on children

Causal inference is a statistical approach that aims to understand and quantify the causal relationship between variables, allowing us to determine the impact of one variable on another while accounting for potential confounding factors. In this study, we chose the Infant Health and Development Program (IHDP) dataset to test whether high-quality early childhood education and medical care will enhance their cognitive and academic ability. We use some common and classic algorithms to achieve this study by calculating the CATE (Conditional Average Treatment Effect) of the dataset; if the result is a negative number, that means early treatment doesn’t improve children’s further ability; if the result is a positive number, that means treatment improves it. After fitting the targeted models, BART(Bayesian Additive Regression Trees) and Metalearners, into this dataset, we found out that all the models will get positive CATE, which means that these early treatments positively affect children’s future development. Also, we can judge which model gives us a more accurate result according to the standard variance of TE(treatment effect). The result of this study will provide us with an insightful idea about whether we should give children some treatments regarding education and medical care and which model is more suitable for this casual inference test.

Visualisations for Bayesian Additive Regression Trees

Tree-based regression and classification has become a standard tool in modern data science. Bayesian Additive Regression Trees (BART) has in particular gained wide popularity due its flexibility in dealing with interactions and non-linear effects. BART is a Bayesian tree-based machine learning method that can be applied to both regression and classification problems and yields competitive or superior results when compared to other predictive models. As a Bayesian model, BART allows the practitioner to explore the uncertainty around predictions through the posterior distribution. In this paper, we present new Visualisation techniques for exploring BART models. We construct conventional plots to analyse a model’s performance and stability as well as create new tree-based plots to analyse variable importance, interaction, and tree structure. We employ Value Suppressing Uncertainty Palettes (VSUP) to construct heatmaps that display variable importance and interactions jointly using colour scale to represent posterior uncertainty. Our new Visualisations are designed to work with the most popular BART R packages available, namely BART, dbarts, and bartMachine. Our approach is implemented in the R package bartMan (BART Model ANalysis).

Early ICD implantation following out-of-hospital cardiac arrest: a retrospective cohort study from the Swedish Registry for Cardiopulmonary Resuscitation

BackgroundIt is unclear whether an implantable cardioverter-defibrillator (ICD) is generally beneficial in survivors of out-of-hospital cardiac arrest (OHCA).ObjectiveWe studied the association between ICD implantation prior to discharge and survival in...

Modelling urban/rural fractions in low- and middle-income countries

Abstract In low- and middle-income countries, household surveys are the most reliable data source to examine health and demographic indicators at the subnational level, an exercise in small area estimation. Model-based unit-level models are favoured for producing the subnational estimates at fine scale, such as the admin-2 level. Typically, the surveys employ stratified 2-stage cluster sampling with strata consisting of an urban/rural designation crossed with administrative regions. To avoid bias and increase predictive precision, the stratification should be acknowledged in the analysis. To move from the cluster to the area requires an aggregation step in which the prevalence surface is averaged with respect to population density. This requires estimating a partition of the study area into its urban and rural components, and to do this we experiment with a variety of classification algorithms, including logistic regression, Bayesian additive regression trees, and gradient boosted trees. Pixel-level covariate surfaces are used to improve prediction. We estimate spatial HIV prevalence in women of age 15–49 in Malawi using the stratification/aggregation method we propose.

Establishing the relationship between wildfire smoke and performance metrics on finished beef cattle in Western Rangelands.

Identifying causal relationships is complicated. Researchers usually overlook causality behind relationships which can generate misleading associations. Herein, we carefully examine the parametric relationship and causality between wildfire smoke exposure and animal performance and behavior metrics over a period of 2 yr in Reno, Nevada. The animals in the 2020 smoke season were grain-finished (n = 12) and grass-finished (n = 12), whereas the animals during the 2021 season were fed under the same diet but finished with either a hormonal implant (n = 9), or without (n = 9). The dataset included daily records of feed intake (FI), body weight (BW), water intake (WI), average daily gain (ADG), and WI behavior (time spent drinking [TSD]; water intake events [WIE]; no-WIE [NWIE]). Variable tree length Bayesian additive regression trees (BART) were utilized to investigate the relationships between air quality index (AQI), particulate matter 2.5 μm (PM2.5) and 10 μm (PM10), NO2, SO2, Ozone, and CO levels in the air (sensors < 1.6 km from animals) with the animal data. Additionally, linear mixed models with a 7-d lag were used to evaluate parametric relationships among the same variables. All statistical analyses were performed on R Statistical Software (R Core Team 2023). Under the linear mixed model with a 7-d lag, significant positive and negative associations were found for all parameters examined (P < 0.05). Negative associations were found between FI, WI, ADG, BW, WIE, NWIE, TSD, and PM2.5 (P < 0.05) for at least one animal group. Positive linear associations between wildfire smoke parameters and the metrics evaluated were more variable and dependent on year, treatment, and smoke parameters. When examining the credible intervals and the variable importance in the BART, relationships were more difficult to identify. However, some associations were found for Ozone, AQI, NO2, CO, and PM10 (P < 0.05). Overall, our results carefully examine the relationship between smoke parameters and cattle performance and present interesting pathways previously unexplored that could guide early culling/finishing of animals to avoid economic losses associated with performance decrease in response to wildfire smoke exposure. Though interesting associations are found under linear mixed models, causality is difficult to establish, which highlights the need for controlled exposure experiments.

Ordered probit Bayesian additive regression trees for ordinal data

Bayesian additive regression trees (BART) is a nonparametric model that is known for its flexibility and strong statistical foundation. To address a robust and flexible approach to analyse ordinal data, we extend BART into an ordered probit regression framework (OPBART). Further, we propose a semiparametric setting for OPBART (semi‐OPBART) to model covariates of interest parametrically and confounding variables nonparametrically. We also provide Gibbs sampling procedures to implement the proposed models. In both simulations and real data studies, the proposed models demonstrate superior performance over other competing ordinal models. We also highlight enhanced interpretability of semi‐OPBART in terms of inference through marginal effects.

Countering “Fake News” Through Public Education and Advertisements: An Experimental Analysis

This paper examines whether proactive efforts to educate people about disinformation through advertisements can successfully increase skepticism towards false headlines or if such efforts do more harm than good by inadvertently increasing belief in false information. We analyze a survey experiment that employed three different advertisements that directly addressed “fake news.” We find that all advertisements were effective at increasing skepticism towards “fake news” headlines. We also find no evidence of backfire effects occurring. However, subsequent analysis using Bayesian additive regression trees (BART) finds significant heterogeneity within these treatment effects. While all advertisements were effective, each ad was effective in different ways despite common themes and content. This suggests a more complicated understanding of the counter-disinformation process and highlights BART’s utility in public opinion research.